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Abstract 

To teach the rigorous skills and knowledge students need to succeed in future college- 
entry courses and workforce training programs, education stakeholders have increasingly 
called for more rigorous curricula, instruction, and assessments. Identifying the critical 
attributes of rigor and measuring its appearance in curricular materials is therefore 
fundamental to progress in this area. By superposing two widely accepted models for 
describing rigor — Bloom's Taxonomy of Educational Objectives and Webb’s Depth-of- 
Knowledge (DOK) model — this article defines cognitive rigor (CR) and introduces the CR 
matrix for analyzing instruction and enhancing teacher lesson planning. Two large-scale 
collections of student work samples analyzed using the CR matrix are presented, 
illustrating the preponderance of curricular items aligned to each cell in the matrix. 

Educators should use the cognitive rigor matrix to align the content in their curricular 
materials to the instructional techniques used in classroom delivery. 

Key words: Bloom's Taxonomy, Webb's depth of knowledge, cognitive rigor, critical 
thinking, enacted curriculum, delivered curriculum 

I ntroduction 

A mainstay for over 50 years, Bloom's Taxonomy helps teachers formulate lessons that practice 
and develop thinking skills over a wide range of cognitive complexity. (Bloom, 1956) Although later 
revised by a team of education researchers headed by Anderson and Krathwohl (2001), the overall intent 
of the taxonomy remains: Categorize questions and activities according to their levels of abstraction. 
However, Bloom's Taxonomy suffers limitations when selecting test items and formulating questioning 
strategies because it uses verbs to differentiate taxonomy levels — many verbs appear at multiple levels 
and do not clearly articulate the intended complexity implied by the taxonomy. A new model of rigor, depth 
of knowledge (DOK), fills this void. The resulting combination of Bloom's Taxonomy and depth of 
knowledge — cognitive rigor — forms a comprehensive structure for defining rigor, thus posing a wide 
range of uses at all levels of curriculum development and delivery. 
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In one imaginary setting, a professional learning community of high school teachers prepares a 
common lesson plan centered on the causes of World War I. Their years of experience and 
professional development have developed considerable skill in associating differing levels of the 
revised Bloom's Taxonomy with their assignment questions. They know that explaining the meaning of 
a metaphor aligns to the analyze level and that if they need an eva/tvafe-level question they can begin 
with the verb critique. So trawling up the (revised) taxonomy levels snares a string of questions that 
will enhance engagement and compel students to examine history a variety of angles. 

The lowest taxonomy level, remember, produces an obvious question: “List four causes of 
WWI.” Although this question will fail to activate higher-order thinking, everyone agrees that it 
will help at least compel students to pay attention to the lesson. 

“Describe each cause in your own words” addresses the understand level of the taxonomy. 
Although the teachers disagree on whether the question truly constitutes high order, all agree 
that responding will intellectually challenge their students more than simply listing the causes. 

The analyze level delivers an enticing possibility: “How did each cause affect the other 
causes?” A similar challenge awaits students responding to the next offering aimed at the fifth 
{evaluate) level: “Rank the causes from most to least important and justify.” 

Not wanting to be outdone, another teacher offers her own contribution: “Suppose the first 
cause never took place; write an essay on what would have happened and why.” An argument 
over the answer breaks out among the teachers over this sixth-order (create) question, 
reminding them of the importance of Bloom's Taxonomy when developing a stimulating and 
engaging curriculum. 

Beginning with Bioom 

Although Bloom originally identified three separate domains of educational activities, most educators 
have trained their attention on the one associated with mental skill — the cognitive (knowledge) 
domain. Anderson and Krathwohl's revision of Bloom's Taxonomy (2001) presented a structure for 
rethinking this domain. (See Table 1 .) Whereas the original cognitive domain possessed only one 
dimension, the revised version encompasses two: Cognitive processes and knowledge. The cognitive 
processes resemble those found in the original taxonomy, but the placement of each level on the 
taxonomy continuum shifted (e.g., evaluation no longer resides at the highest level) and includes 
expanded and clarified descriptions for analyzing educational objectives. The revised descriptors 
consider both the processes (the verbs) and the knowledge (the nouns) used to articulate educational 
objectives. This restructuring of the original taxonomy recognizes the importance of the interaction 
between content (characterized by factual, conceptual, procedural, and metacognitive knowledge) and 
thought processes. 

The teachers in the professional learning community look back on their meeting with 
satisfaction — each question they formulated addressed distinct thinking processes. Later, 
however, these teachers discover that Bloom's Taxonomy offers insufficient guidance in 
formulating assessment and instructional delivery strategies. With no natural tie between the 
taxonomy levels and the depth of understanding required to respond to each question, their 
assessment strategies begin to fall back on traditional crude rules of thumb and gut feel. As 
one teachers puts it, “Not all of the questions residing at the same level of Bloom's Taxonomy 
offer my students the same mental challenge.” Complicating matters even further, many verbs 
such as compare and explain appear at multiple levels in the taxonomy. 

Later, some of the teachers begin to discover a new measure of rigor in the literature that bridges this 
deficiency: Depth of knowledge. 
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Table 1 

Comparison of descriptors associated with the cognitive process dimensions of Bloom’s original 
taxonomy (1956) and the revised Bloom’s Taxonomy of Anderson and Krathwohl (2001). 

Bloom’s Taxonomy (1956) Revised Bloom Process Dimensions (2005) 



Knowledge 

Define, duplicate, label, list, memorize, name, order, 
recognize, relate, recall, reproduce, state 

Comprehension 

Classify, describe, discuss, explain, express, identify, 
indicate, locate, recognize, report, restate, review, 
select, translate 



Application 

Apply, choose, demonstrate, dramatize, employ, 
illustrate, interpret, practice, schedule, sketch, solve, 
use, write 

Analysis 

Analyze, appraise, calculate, categorize, compare, 
criticize, discriminate, distinguish, examine, 
experiment, explain 



Synthesis 

Rearrange, assemble, collect, compose, create, 
design, develop, formulate, manage, organize, plan, 
propose, set up, write 

Evaluation 

Appraise, argue, assess, choose, compare, defend, 
estimate, explain, judge, predict, rate, core, select, 
support, value, evaluate 



Remember 

Retrieve knowledge from long-term memory, 
recognize, recall, locate, identify 

Understand 

Construct meaning, clarify, paraphrase, represent, 
translate, illustrate, provide examples, classify, 
categorize, summarize, generalize, infer a logical 
conclusion (such as from examples given), predict, 
match similar ideas, explain, compare/contrast, 
construct models (e.g., cause-effect) 

Apply 

Carry out or use a procedure in a given situation; 
carry out (apply to a familiar task) or use (apply) 
to an unfamiliar task 

Analyze 

Break into constituent parts, determine how parts 
relate, differentiate between relevant and 
irrelevant, distinguish, focus, select, organize, 
outline, find coherence, deconstruct (e.g., for bias 
or point of view) 

Evaluate 

Judge based on criteria, check, detect 
inconsistencies or fallacies, judge, critique 

Create 

Combine elements to form a coherent whole, 
reorganize elements into new 
patterns/structures, generate, hypothesize, 
design, plan, construct, produce for a specific 
purpose 
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Webb's DOK model 

Depth of knowledge (DOK) forms another important perspective of cognitive complexity. The best- 
known work in this area, that of Norman Webb (1997, 1999), compelled states to rethink the meaning 
of test alignment to include both the content assessed in a test item and the depth to which we expect 
students to demonstrate understanding of that content. In other words, the complexity of both the 
content (e.g., interpreting literal versus figurative language) and the required task (e.g., solving routine 
versus non-routine problems) both define each DOK level shown in Table 2. 

Table 2 

Webb’s depth-of-knowledge (DOK) levels (Webb 1997, 1999) 

Level Description 

DOK-1 Recall & Reproduction — Recall a fact, term, principle, or concept; perform a routine procedure. 

DOK-2 Basic Application of Skills/Concepts — Use information, conceptual knowledge; select 

appropriate procedures for a task; perform two or more steps with decision points along the 
way; solve routine problems; organize or display data; interpret or use simple graphs. 

DOK-3 Strategic Thinking — Reason or develop a plan to approach a problem; employ some 
decision-making and justification; solve abstract, complex, or non-routine problems, 
complex. (DOK-3 problems often allow more than one possible answer.) 

DOK-4 Extended Thinking — Perform investigations or apply concepts and skills to the real world that 
require time to research, problem solve, and process multiple conditions of the problem or task; 
perform non-routine manipulations across disciplines, content areas, or multiple sources. 



Although related through their natural ties to the complexity of thought. Bloom's Taxonomy and Webb’s 
DOK model differ in scope and application. Bloom's Taxonomy categorizes the cognitive skills required 
of the brain when faced with a new task, therefore describing the type of thinking processes necessary 
to answer a question. The DOK model, on the other hand, relates more closely to the depth of content 
understanding and scope of a learning activity, which manifests in the skills required to complete the 
task from inception to finale (e.g., planning, researching, drawing conclusions). 

Today, interpreting and assigning intended DOK levels to both the standards and the related assessment 
items form critical components of any alignment analysis. Educators have applied Webb’s DOK levels 
across all content areas (Hess, 2004, 2005a, 2005b, 2006a, 2006b; Petit & Hess, 2006). Many states and 
districts employ DOK to designate the depth and complexity of state standards to align the state’s large- 
scale assessments or to revise existing standards to achieve higher cognitive levels for instruction. 
Consequently, teachers need to develop the ability to design instruction and create units of curriculum and 
classroom assessments for a greater range of cognitive demand. 

Teachers in the professional learning community slowly begin to assimilate depth of knowledge 
in their lesson planning, as they find identifying the DOK levels of questions an effective step in 
formulating questioning strategies and optimizing assessments. One teacher notes that DOK 
levels do not correlate to her notion of “difficulty.” Another agrees, pointing out that students will 
struggle more to remember all four causes than only one, but both activities reside at the DOK- 
1 level. 

The teachers soon begin to see how depth of knowledge fits in lesson planning. “A DOK-3 or 
DOK-4 learning activity requires a great deal of assimilation of information. My students will 
need to converse at length in groups to reinforce what they learned earlier and will need more 
wait time and engagement with the content.” Another teacher chimes in: “I can see how 
categorizing my questions according to depth of knowledge helps in creating assessments. Too 
many DOK-1 questions and I'm not sure I have truly measured their conceptual understanding 
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of the content.” The teachers conclude that a reasonable blend of DOK-1 , DOK-2, and DOK-3 
questions comprises a reasonable assessment, relegating DOK-4 questions to in-class 
projects. They also agree to apply DOK-3 and DOK-4 activities toward content they expect their 
students to know more deeply. 

As these teachers discovered, identifying the DOK levels of questions in tests and class assignments 
helps articulate how deeply students must understand the related content to complete necessary 
tasks. As examples, students need a greater depth of understanding to explain how or why a concept 
or rule works (DOK-2), to apply it to real-world phenomena with justification or supporting evidence 
(DOK-3), or to integrate a given concept with other concepts or other perspectives (DOK-4). 

Cognitive rigor and the CR matrix 

Because no simple one-to-one correspondence relates Bloom's Taxonomy and depth of knowledge, 
Hess (2006b) superposed them. The resulting cognitive rigor (CR) matrix in Table 3 vividly connects, 
yet clearly distinguishes, the two schemata, allowing educators to examine the rigor associated with 
tasks that might seem at first glance comparable in complexity. (See appendix.) The cognitive rigor 
matrix soon found use in states just beginning to appreciate the role cognitive complexity played in 
test design and item development (Hess, 2006a, 2006b). 

Although extending a pattern in mathematics may not look anything like distinguishing fact from 
opinion in English language arts, the two tasks reside in the same cell of the CR matrix and therefore 
share many common features: Both evoke similar thought processes and require similar instructional 
and assessment strategies. For curriculum analysis purposes, the CR matrix effectively categorizes 
such learning activities that appear prominently in curriculum and instruction. For example, the rote 
completion of single-step mathematical routines, often derided by the moniker “plug and chug,” 
resides within the [DOK-1 , Bloom-3] cell of the CR matrix. Using the CR matrix to plot the cognitive 
rigor of mathematics assignments, as shown in Fig. 1 , a researcher or practitioner can discover the 
extent to which the curriculum targets this level of cognitive rigor compared to the [3, 2] cell, which 
similarly requires application of learned concepts to new situations (Bloom-3) but requires some 
decision-making to perform (DOK-2). 

Research results applying the CR matrix 

In two large-scale studies of the enacted (i.e., taught) mathematics and English language arts 
curricula, teachers from 200 Nevada and Oklahoma public schools submitted for analysis over 
200,000 samples of student work, which included homework samples, tests, quizzes, and worksheets. 
(The Standards Company LLC, 2008a, 2008b). Curriculum specialists analyzed each item on each 
work sample and, using the CR matrix, assigned to each collected sample its overall DOK level and 
the highest Bloom's Taxonomy level appearing on the sample. The CR density plots in Figure 1 
illustrate results from the two studies, displaying the percentage of assignments as a color shade. 

(The studies encompass hundreds of such plots disaggregated according to grade level, subject area/ 
course, socioeconomic status, and so on; the two plots shown in Figure 1 merely represent cumulative 
examples for each subject area.) 

Results for English language arts indicate a preponderance of assignments correlating to the [2, 2] cell 
of cognitive rigor. (The two coordinates denote the levels of DOK and Bloom's Taxonomy, 
respectively.) Mathematics assignments, on the other hand, heavily sampled the [1,1] and [1 , 3] cells. 
Although assignments associated with the latter, such as solving algebraic equations and non-rote 
arithmetic, help students practice crucial numeracy and fluency skills in mathematics, the result 
nonetheless may point to an over-reliance on teaching straightforward applications of routine steps. 
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Figure 1: Density plots comparing the cognitive rigor of student work assignments collected from 
205 schools across two states. (The Standards Company LLC 2008a, The Standards Company 
LLC 2008b). Although the study encompassed all public school grade levels, the above plots only 
show third-grade results. Thin cells lined along the left and bottom display cumulative 
percentages for each row or column. The English language arts data comprises 12,060 samples 
of student work; mathematics comprises 8,428. Results that round to 0 remain unlabeled for 
clarity; as a result, percentages might not always sum to 100%. 



Figure 1 also indicates that the Bloom's Taxonomy levels reached by both subject areas appeared 
similar, but mathematics sampled the lowest (DOK-1) levels of depth of knowledge to a greater extent, 
indicating that the mathematics curriculum delivered by teachers may rely on a shallower 
understanding of the subject than English language arts. 

Discussion 

Students learn skills and acquire knowledge more readily when they can transfer their learning to new 
or more complex situations, a process more likely to occur once they have developed a deep 
understanding of content (National Research Council, 2001). Therefore, ensuring that a curriculum 
aligns to standards alone will not prepare students for the challenges of the twenty-first century. 
Teachers must therefore provide a// students with challenging tasks and demanding goals, structure 
learning so that students can reach high goals, and enhance both surface and deep learning of 
content (Hattie, 2002). 

Both Bloom's Taxonomy and Webb's depth of knowledge therefore serve important functions in 
education reform at the state level in terms of standards development and assessment alignment. 
Because cognitive rigor encompasses the complexity of content, the cognitive engagement with that 
content, and the scope of the planned learning activities, the CR matrix can enhance instructional and 
assessment practices at the classroom level as well. Superposing the two cognitive complexity 
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measures produces a means of analyzing the emphasis placed on each intersection of the matrix. As 
educators become more skilled at recognizing cognitive rigor and analyzing its implications for 
instruction and assessment, they can enhance learning opportunities for all students and across all 
subject areas and grade levels. Because students need exposure to novel and complex activities 
every day, schools in the twenty-first century should prepare students by providing them with a 
curriculum that spans a wide range of the cognitive rigor matrix. 
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Table 3 

Cognitive rigor (CR) matrix with curricular examples. 



Webb’s Depth-of-Knowledge Levels 

Level 3 



Revised Bloom’s Level 1 Level 2 Strategic Thinking/ Level 4 

Taxonomy levels Recall and Reproduction Skills and Concepts Reasoning Extended Thinking 



Remember 

Retrieve knowledge from long-term 
memory, recognize, recall, locate, 
identify 


Recall, recognize, locate basic 
facts, ideas, principles 

Recall or identify conversions: 
between units of measure 

Identify facts/details in texts 








Understand 

Construct meaning, clarify, para- 
phrase, represent, translate, illus- 
trate, give examples, classify, cat- 
egorize, summarize, generalize, in- 
fer a logical conclusion, predict, 
compare/contrast, match like ideas, 
explain, construct models 


Compose/decompose numbers 
Evaluate an expression 
Locate points on a grid 
Symbolize math relationships 
Write simple sentences 
Describe/explain how or why 


Specify and explain relationships 
Give non-examples/examples 
Make and record observations 
Summarize results, concepts, ideas 
Infer or predict from data or texts 
Identify main ideas 


Explain, generalize, or connect 
ideas using supporting evidence 

Explain phenomena in terms of 
concepts 

Write full composition to meet spe- 
cific purpose 

Identify themes 


Explain how concepts or ideas spe- 
cifically relate to other content do- 
mains or concepts 

Develop generalizations of the res- 
ults obtained or strategies used and 
apply them to new problem situ- 
ations 


Apply 

Carry out or use a procedure in a 
given situation; carry out (apply to a 
familiar task), or use (apply) to an 
unfamiliar task 


Follow simple/routine procedures 
Solve a one-step problem 
Calculate, measure, apply a rule 
Apply an algorithm or formula 

Represent in words or diagrams a 
concept or relationship 

Apply rules or use resources to 
edit spelling and grammar 


Select a procedure according to task 
needed and perform it 

Solve routine problem applying mul- 
tiple concepts or decision points 

Retrieve information from a graph 
and use it solve a multi-step problem 

Use models to represent concepts 

Write paragraph using appropriate 
organization, text structure 


Use concepts to solve non-routine 
problems 

Design investigation for a specific 
purpose or research question 

Conduct a designed investigation 

Use reasoning, planning, and 
evidence 

Revise final draft for meaning or 
progression of ideas 


Select or devise an approach 
among many alternatives to solve a 
novel problem 

Conduct a project that specifies a 
problem, identifies solution paths, 
solves the problem, and reports 
results 

Illustrate how multiple themes (his- 
torical, geographic, social) may be 
interrelated 


Analyze 

Break into constituent parts, de- 
termine how parts relate, differenti- 
ate between relevant-irrelevant, 
distinguish, focus, select, organize, 
outline, find coherence, deconstruct 
(e.g., for bias or point of view) 


Retrieve information from a table 
or graph to answer a question 

Identify or locate specific informa- 
tion contained in maps, charts, 
tables, graphs, or diagrams 


Categorize, classify materials 
Compare/ contrast figures or data 
Select appropriate display data 
Extend a pattern 
Identify use of literary devices 
Identify text structure of paragraph 


Compare information within or 
across data sets or texts 

Analyze and draw conclusions 
Generalize a pattern 
Organize/interpret data 
Analyze author’s craft or viewpoint 


Analyze multiple sources of evid- 
ence or multiple works by the same 
author, or across genres 

Analyze complex/abstract themes 

Gather, analyze, and organize in- 
formation 

Analyze discourse styles 


Evaluate 

Make judgments based on criteria, 
check, detect inconsistencies or fal- 
lacies, judge, critique 






Cite evidence and develop a logic- 
al argument for concepts 

Describe, compare, and contrast 
solution methods 

Verify reasonableness of results 
Justify conclusions made 


Gather, analyze, and evaluate rel- 
evancy and accuracy 

Draw and justify conclusions 

Apply understanding in a novel way, 
provide argument or justification for 
the application 


Create 

Reorganize elements into new pat- 
terns/structures, generate, hypo- 
thesize, design, plan, construct, 
produce 


Brainstorm ideas, concepts, or 
perspectives related to a topic or 
concept 


Generate conjectures or hypotheses 
based on observations or prior 
knowledge 


Synthesize information within one 
source or text 

Formulate an original problem 

Develop a complex model for a 
given situation 


Synthesize information across mul- 
tiple sources or texts 

Design a model to inform and solve 
a real-world, complex, or abstract 
situations 





























